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(57) ABSTRACT 

MPEG-J is a collection of Java application programming 
interfaces (APIS) with which applications can be developed 
to interact with the platform and the content. In the context 
of MPEG-J, the platform is a device like a set-top box or a 
PC with Java packages conforming to a well-defined Java 
platform. The Java-based application consists of Java byte 
code, which may be available from a local source, like a hard 
disk, or it may be loaded from a remote site over a network. 
The MPEG-J Java byte code will be available as a separate 
elementary stream. The MPEG-4 system is the "Presentation 
engine" of MPEG-J. MPEG-J provides programmatic con- 
trol through an "Application engine" which enhances the 
MPEG-4 browser by providing added interactive capability. 

6 Claims, 5 Drawing Sheets 



421 



BUFFER 




CLASS 


V 


LOADER 



122^37" 
BACK 1 — 

CHANNEL 



DEMUX 



104 



CHANNEL 



102 




=X | Birs oecomTMh scene graph") :: 



114 



DEC BUFFER 1 h * H MEDIA DEC 1 hHCOMP BUFFER!] ^ 
r .~ ( 132 r— 152 r- 172 
DEC BUFFER 1 rH MEDIA DEC 2 HcOMP BUFFER 2> ^ 

S • s : ^ : 

130 .... : r — 150 : 170 : 



110 



T 

114 



DEC BUFFER n 
130 



} H MEDIA DEC n 
132 150 



¥ 

152 



COMP BUFFER 

~\ 

170 



172 





-180 


COMPOSITOR 




AND 
RENDERER 




J" 

182 





07/25/2004, EAST Version: 1.4.1 



U.S. Patent 



Oct. 7, 2003 



Sheet 1 of 5 



US 6,631,403 Bl 




07/25/2004, EAST Version: 1.4.1 



U.S. Patent 



Oct. 7, 2003 



Sheet 2 of 5 



US 6,631,403 Bl 



FIG. 2 



220 



MPEG-J APP 



20 



MPEG-J RUN-TIME ENVIRONMENT 



22 



JAVA 
VIRTUAL 
MACHINE 



] 0VQ.... 



j QVO.IO. 



26 



java.lang 



MPEG-J APIs 



24 



MPEG-J APPLICATION ENGINE 



200 



7 



PRESENTATION 
ENGINE 



182 



EXECUTION 
ENGINE 



100 



190 



MULTI-MEDIA 0ATA PROCESS ING SYSTEM 

7 



10 



07/25/2004, EAST Version: 1.4.1 



U.S. Patent Oct. 7, 2003 Sheet 3 of 5 



US 6,631,403 Bl 




07/25/2004, EAST Version: 1.4.1 



U.S. Patent 



Oct. 7, 2003 



Sheet 4 of 5 



US 6,631,403 Bl 




07/25/2004, EAST Version: 1.4.1 



U.S. Patent Oct. 7, 2003 Sheet 5 of 5 US 6,631,403 Bl 




07/25/2004, EAST Version: 1.4.1 



US 6,6: 

1 

ARCHITECTURE AND APPLICATION 
PROGRAMMING INTERFACES FOR JAVA- 
ENABLED MPEG-4 (MPEG-J) SYSTEMS 

This non-provisional application claims the benefit of 
U.S. provisional application No. 60/085,001, entitled 
"MPEG-J Architecture and API (V0.4)", filed on May 11, 
1998, and which is incorporated herein by reference in its 
entirety. The applicants of the provisional application are 
Keith Deutsch, Gerard M. Fernando, Atul Puri, Robert L, 
Schmidt, Pallavi Shah and Viswanathan Swaminathan. 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

This invention is directed to multimedia data storage, 
transmission and compression systems and. methods. In 
particular, this invention is directed to systems and methods 
that implement the MPEG-J multimedia data storage, trans- 
mission and compression standards. This invention is also 
directed to control systems and methods that allow for 
graceful degradation and enhanced functionality and user 
interactivity of MPEG-4 systems. 

2. Related Art 

The need for interoperability, guaranteed quality and 
performance and economies of scale in chip design, as well 
as the cost involved in content generation for a multiplicity 
of formats, has lead to advances in standardization in the 
areas of multimedia coding, packetization and robust deliv- 
ery. In particular, the International Standards Organization 
Motion Picture Experts Group (ISO MPEG) has created a 
number of standards, such as MPEG-1, MPEG-2, MPEG-4 
and MPEG-J to standardize bitstream syntax and decoding 
semantics for coded multimedia. 

In MPEG-1 systems and MPEG-2 systems, the audio- 
video model was very simple, where a given elementary 
stream covered the entire scene. In particular, MPEG-1 
systems and MPEG-2 systems were only concerned with 
representing temporal attributes. Thus, there was no need to 
represent spatial attributes in a scene in MPEG-1 systems 
and MPEG-2 systems. 

The success of MPEG-1 and MPEG-2, the bandwidth 
limitations of the Internet and other distributed networks and 
of mobile channels, the flexibility of distributed network- 
based data access using browsers, and the increasing need 
for interactive personal communication has opened up new 
paradigms for multimedia usage and control. The MPEG-4 
standard addresses coding of audio-visual information in the 
form of individual objects and a system for combining and 
synchronizing playback of these objects. 

MPEG-4 systems introduced audio-video objects, requir- 
ing that the spatial attributes in the scene also need to be 
correctly represented. Including synthetic audio-video con- 
tent in MPEG-4 systems is a departure from the model of 
MPEG-1 systems and MPEG-2 systems, where only natural 
audio- video content representation was addressed. MPEG-4 
systems thus provide the required methods and structures for 
representing synthetic and natural audio-video information. 
In particular, MPEG-4 audio- video content has temporal and 
spatial attributes that need to be correctly represented at the 
point of content generation, i.e., during encoding, and that 
also need to be correctly presented at the player/decoder. 
Because the MPEG-4 player/decoder also allows for limited 
user interactivity, it should more properly be referred to as 
an MPEG-4 browser. 

Correctly representing temporal attributes in MPEG-4 
systems is essentially no different than in MPEG-lsystems 
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and MPEG-2 systems. For these earlier standards, the tem- 
poral attributes were used to synchronize the audio portions 
of the data with the video portions of the data, i.e., audio- 
video synchronization such as lip-synchronization, and to 
5 provide system clock information to the decoder to help 
buffer management. Because significantly more diverse 
types of elementary streams can be included in MPEG-4 
systems, representing temporal attributes is more complex. 
But, as mentioned earlier, the fundamental methods for 
representing temporal attributes in MPEG-4 systems is 
essentially the same as for MPEG-1 systems and MPEG-2 
systerhs. 

In the MPEG-1 systems and MPEG-2 systems standards, 
the specifications extend monolithically from the packetiza- 
tion layer all the way to the transport layer. For example, the 

15 MPEG-2 systems Transport Stream specification defined the 
packetization of elementary streams (Le., the PES layer) as 
well as the Transport layer. With MPEG-4 systems, this 
restriction has been relaxed. The transport layer is not 
defined normative ly, as the transport layer is very applica- 

20 tion specific. It is left to other standards setting bodies to 
define the transport layer for their respective application 
areas. One such body is the Internet Engineering Task Force 
(IETF), which will define standards for transporting 
MPEG-4 streams over the Internet. 

25 

Representing spatial information in MPEG-4 systems is 
carried out using a parametric approach to scene description. 
This parametric approach uses the Virtual Reality Modeling 
Language (VRML). The Virtual Reality Modeling Language 
allows spatial and temporal relationships between objects to 
be specified, and allows description of a scene using a scene 
graph approach. 

The scene description defines one or more dynamic 
properties of one or more audio and video objects. However, 
35 in MPEG-4 systems, the Virtual Reality Modeling Language 
has been extended to provide features otherwise missing 
from Virtual Reality Modeling Language. 

MPEG-4 uses a binary representation, Binary Format for 
Scene (BIFS), of the constructs central to VRML and 
40 extends VRML in many ways to handle real-time audio/ 
video data and facial/body animation. The key extensions to 
Virtual Reality Markup Language for MPEG-4 systems 
involve streaming, timing and integrating 2D and 3D 
objects. These extensions are all included in the Binary 
45 Format for Scene (BIFS) specification. 

FIG. 1 outlines one exemplary embodiment of a MPEG-4 
systems player, which is also referred to as a "Presentation 
Engine" or an "MPEG-4 browser". The main components on 
the main data path are the demultiplexer layer, the media 
50 decoders, and the compositor/renderer. Between these three 
sets of components there are decoder buffers and composi- 
tion buffers, respectively. The MPEG-4 systems decoder 
model has been developed to provide guidelines for platform 
developers. The binary format for scene data is extracted 
55 from the demultiplexer layer, and it is used to construct the 
scene graph. 

Using application programming interfaces (APIs) has 
been long recognized in the software industry as a way to 
achieve standardized operations and functions over a num- 

60 ber of different types of computer platforms. Typically, 
although operations can be standardized via definition of the 
API, the performance of these operations may still differ on 
various platforms, as specific vendors with interest in a 
specific platform may provide implementations optimized 

65 for that platform. 

To enhance the features of VRML and to allow program- 
matic control, DimensionX has released a set of APIs known 
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as Liquid Reality. Recently, Sun Microsystems has This invention provides systems and methods that use a 

announced an early version of Java3D, an API specification combination of MPEG-4 media and safe executable code so 

that supports representing synthetic audiovisual objects as a that content creators can embed complex control mecha- 

scene graph. Sun Microsystems has also released the Java nisms with in their media data to intelligently manage the 

Media Framework Player API, a framework for multimedia 5 operation of the audio-visual session, 

playback. invention separately provides systems and methods 

SUMMARY OF THE INVENTION for implementing the MPEG-J video data storage, compres- 

As noted above, when coded multimedia is used for sion and decodin S standards, 

distributed networked and local networked applications on a 1Q invention separately provides an improved MPEG-J 

multimedia data processing system, such as a personal architecture. 

computer, a number of situations may arise. First, the This invention additionally provides an improved 

bandwidth for networked access of multimedia may be MPEG-J architecture having improved structure, modularity 

either limited or time-varying, requiring transmission of and organization. 

only the most significant information, followed by transmit- 15 This invention separately provides an MPEG-J applica- 

ting additional information as more bandwidth becomes tion engine that allows for graceful degradation of MPEG-4 

available. content in view of limited processing, memory or bandwidth 

Second, regardless of the bandwidth available, the client, resources, 

i.e., the multimedia data processing system, decoding the Thi s invention separately provides an MPEG-J applica- 

transmitted information may be limited in processing and/or 20 tion engine tnat allows for enna nced functionality of and 

memory resources. Furthermore, these resources may be uscr interactivity with MPEG-4 content, 

time-varying. Third, a multimedia user may require highly ^ .^.^ tel ides aw>licaUon ^g^. 

interactive nonlinear browsing and playback. This is not interfaces for MPEG-J. 

unusual, because significant amounts of textual content on 0 

distributed networks, such as the Internet, are capable of « New paradigms in communication software and 

being browsed using hyperlinked features and because this networking, such as that offered by the Java™ language, 

is also expected to be true for presentations employing offer Qew opportunities for flexibility, adaptivity and user 

coded audio-visual objects. The parametric MPEG-4 system interaction. For instance, the advent of the Java™ language 

may only be able to deal with the these situations in a very offers networking and platform independence critical to 

limited way. For example, when the parametric MPEG-4 30 downloading and executing of applets, such as, for example, 

system is incapable of decoding or presenting all of the Java classes, on a client system from a server system stonng 

coded audio-visual objects, the parametric MPEG-4 system the a PP lets - Depending on the design of the applet, either a 

may respond by dropping those objects or temporal occur- sin S lc access t0 data stored °° the ^ ma y ^ needed 

rences of those objects. However, this results in choppy and a11 the necessary data may be stored on the client, or 

audio-visual presentations. Further, MPEG-4 may not offer 35 scvcral acccsses ma V be nccded * ^ Partial access 

any sophisticated control to the user to allow the user to deal desi § a 1S used to reduoe stora g e space and ume needed for 

with these situations. startup. This partial access design is referred to as streamed 

To get around the limitations of this known parametric playback. 

MPEG-4 system, another known implementation of the This invention provides a collection of Java API's with 

MPEG-4 standard is a programmatic MPEG-4 system. U.S. 40 which applications can be developed to interact with a data 

patent application Ser. No. 09/055,934, incorporated herein processor and content. According to this invention, MPEG-J 

by reference, discloses such a programmatic MPEG-4 sys- is a Java™-enabled set of standards that define the file 

tem. This programmatic MPEG-4 system includes a set of organization, storage and compression of video data 

defined application programming interfaces (APIs) for streams. In the context of MPEG-J according to this 

media decoding, user functionalities and authoring. These 45 invention, the data processor can be implemented as a 

application programming interfaces can be invoked by client set - t0 P bo * or a pc ^ Java packages conforming to a 

applications. This programmatic MPEG-4 system allows a well-defined Java platform. The Java-based application 

number of enhanced real-time and other functions in includes Java byte code, which may be available from a 

response to user inputs, as well as graceful degradation in local source, like a hard disk, or which may be loaded from 

the face of limited system resources available to MPEG-4 50 a remote site over a network. As indicated above, the term 

clients . * "MPEG-4 browser" refers to the MPEG-4 system. MPEG-J 

The 'incorporated 934 application discloses standardized adds programmatic controlto the MPEG-4 system, through 

interfaces for MPEG-4 playback and browsing under user an "Application engine . The Application Engine enhances 

control, as well as one type of response to time-varying local the Presentation Engine by providing added interactive 

and networked resources. These interfaces facilitate adap- 55 capability. TTie MPEG-J Java byte code will be available to 

tation of coded media data to immediately available terminal ^ MPEG * J Application engine as a separate elementary 

resources. These interfaces also facilitate interactivity stream. 

expected to be sought by users, either directly as a func- The improved architecture and application programming 

tionality or indirectly embedded in audiovisual applications interfaces (APIs) of MPEG-J according to this invention 

and services expected to be important in the future. 6 0 allow selective media decoding facilitating graceful degra- 

The incorporated 934 application also discloses an inter- Nation to varying resources of a client, as well as improved 

facing method in the form of a robust application program- functionalities as required in interactive user applications, 

ming interface specification including a visual decoding In one potential use of MPEG-J, a content provider 

interface, a progressive interface, a hot object interface, a designs all of the MPEG-J content, i.e., the MPEG-J data 

directional interface, a trick mode, a transparency interface, 65 stream. This use of the MPEG-J standard is desirable for 

and a stream editing interface. These interfaces facilitate a content providers, and requires only incremental updates to 

substantial degree of adaptivity. the MPEG-J data stream. In this case, any changes to the 
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MPEG -J data stream can be done using binary format for 
scene (BIFS) updates. In another potential use of MPEG-J, 
the client dynamically controls the displayed video scene 
generated from the MPEG-J data stream. This use of the 
MPEG-J standard desirable for set top manufacturers. 
However, change in non-updatable nodes of the MPEG-J 
data stream may not be possible. 

MPEG-J will eventually need to serve both of the poten- 
tial uses. Since the first potential use is more deterministic 
than the second potential use, the systems and methods of 
this invention fully implement the first potential use. The 
systems and methods of this invention provide the hooks to 
partially implement the second potential use. In particular, in 
one exemplary embodiment of the systems and methods of 
this invention, the MPEG-J scene graph capabilities are 
always based on the tightly-integrated, i.e., content- 
provider-oriented, model. 

In one exemplary embodiment of the application pro- 
gramming interfaces of this invention, the MPEG-J appli- 
cation programming interface (API) is not a single applica- 
tion programming interface, but rather is a collection of 
application programming interfaces (APIs) that address 
various interfaces for a flexible MPEG-4 system. In one 
exemplary embodiment, the MPEG-J application program- 
ming interfaces are implemented using the Java language. 
The application programming interfaces include one or more 
of an Application Manager API; a SceneGraph Manager 
API; a Resource Manager API; an Interactivity Manager 
API; a Media Decoders API; a Functionality API; a Net- 
working API; and a Device API. In one exemplary embodi- 
ment of the application programming interfaces of this 
invention, one or more of these application programming 
interfaces are implemented as object-oriented-programming 
object classes. The object classes are organized into various 
packages. In particular, various objects of one or more of the 
application programming interfaces are collected into a 
single package. 

These and other features and advantages of this invention 
are described in or are apparent from the following detailed 
description of various exemplary embodiments of the sys- 
tems and methods according to this invention. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Various exemplary embodiments of this invention will be 
described in detail, with reference to the following figures, 
wherein: 

FIG. 1 is a functional block diagram of one exemplary 
embodiment of an MPEG-4 browser; 

FIG. 2 is a functional block diagram of one exemplary 
embodiment of the components of an MPEG-J operating 
environment according to this invention; 

FIG. 3 is a functional block diagram showing in greater 
detail a first exemplary embodiment of the MPEG-J appli- 
cation engine of FIG. 2 according to this invention; 

FIG. 4 is a functional block diagram showing in greater 
detail a second exemplary embodiment of the MPEG-J 
application engine of FIG. 2 according to this invention; and 

FIG. 5 is a functional block diagram showing in greater 
detail a third exemplary embodiment of the MPEG-J appli- 
cation engine of FIG. 2 according to this invention. 

DETAILED DESCRIPTION OF THE 
PREFERRED EMBODIMENTS 

FIG. 1 is a functional block diagram outlining one exem- 
plary embodiment of a MPEG-4 browser or presentation 
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engine 100. As shown in FIG. 1, the presentation engine 100 
includes a demultiplexer layer 110, a Delivery Multimedia 
Integration Framework (DMIF) subsystem 120, one or more 
decoder buffers 130, a Binary Format for Scene (BIFS) 

5 decoder 140, one or more media decoders 150, a scene graph 
controller 160, one or more compositor buffers 170, and a 
compositor/renderer 180. In particular, audio- video data is 
input to the demultiplexer layer 120 over a channel signal 
line 102, while handshake data is output by the demulti- 

10 plexer layer 120 over a backchannel signal line 104. 

The demultiplexer layer 110 is connected to the delivery 
multimedia integration framework subsystem 120 over a 
data channel 122. The demultiplexer layer 110 is also 
connected to the binary format for scene decoder 140 over 

15 a data channel 112 and to each decoder buffer 130 over a 
data channel 114. The binary format for scene decoder 140 
is connected to the scene graph controller 160 over a data 
channel 142. Each decoder buffer 130 is connected to a 
corresponding media decoder 150 over a data channel 132. 

20 Each media decoder 150 is connected to a corresponding 
compositor buffer 170 over a dedicated data channel 152. 
Each compositor buffer 170 is connected to the compositor/ 
Tenderer 180 over a data signal channel 172. The 
compositor/renderer 180 is connected over a data channel 

25 182 to a downstream processing subsystem, such as a 
display. 

The scene graph controller 160 outputs control signals to 
the compositor/renderer 180 over a control channel 162. The 
scene graph controller 160 also outputs control signals to 

30 each media decoder 150 over the channel or channels 162. 
In operation, scene content in the form of MPEG-4 
encoded audio-video streams is input to the demultiplexer 
layer 110 over the channel signal line 102, The encoded 

35 audio -video streams include a scene graph stream. The 
demultiplexer layer 110 uses the backchannel signal line 104 
to acknowledge receipt of data, or to request data be resent, 
and, in a client-server environment, sends commands back 
to the server, and the like. The demultiplexer layer 110 then 

4Q demultiplexes the encoded audio-video streams based on 
data input from the delivery multimedia integration frame- 
work subsystem 120 over the data channel 122, and outputs 
the demultiplexed scene graph stream, comprising encoded 
binary format for scene data, to the binary format for scene 

45 decoder 140 over the data channel 112 and each of the other 
encoded audio-video streams an appropriate one of the 
decoder buffers 130 over a corresponding one of the data 
channels 114. The binary format for scene decoder 140 
decodes the encoded binary format for scene data and 

5Q outputs it to the scene graph controller 160 over the data 
channel 142. 

The scene graph controller 160 inputs the decoded binary 
format for scene data and generates a scene graph from the 
input binary format for scene data. The scene graph defines 

55 which objects will appear in the rendered scene, based on the 
binary format for scene data associated with each node of the 
generated scene graph. The scene graph controller 160, 
based on the generated scene graph, outputs decoding con- 
trol signals over the channel or channels 162 to one or more 

60 of the media decoders 150. 

Based on the decoding control signals from the scene 
graph controller 160, one or more of the media decoders 150 
inputs encoded audio -video data appropriate for that media 
decoder 150 from the corresponding decoder buffer 130 over 

65 the corresponding data channel 132. Each such media 
decoder 150 decodes the input audio -video data to generate 
one or more audio or video objects. The decoded audio 
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objects or video objects are output by that media decoder environment. As shown in FIG. 2, a multimedia data pro- 

150 to the corresponding compositor buffer 170 over the cessing system 10 includes an MPEG-J application engine 

corresponding data channel 152. 200, the MPEG -4 browser or presentation engine 100 and 

The compositor buffers 170 store the decoded audio and ^waifon tn f^ 0 ' ^ MPEG : J ,* PpH ? ati °° 

video objects generated by the media decoders 150. The 5 200 includes an MPEG-J a^lication 220 and an MPEG-J 

• i j 1 oa l j i • .1-1 run time environment 20. The MPEG-J run time environ- 

compositor/renderer 180, based on rendering con rol signals ^ ^ Java ^ ^ a 

output by the scene graph controller ^160 over the control of MpEGJ haiiion prograinming interfaces (MPEG-J 
channel 164, inputs vanous audio and video objects stored ^ 24> ag ^ u as the F stan ^ dard Java syslem files 26 . 
m the compositor buffers 170 over the data channels 172 and ^ MpE(M ms and methods q£ ^ 
composes the input audio and video object into a scene. In io and mn applicatio[1 programs that ^ tne behavior 
particular, the scene graph controller 160 generates the 0 f the MPEG-4 browser or presentation engine, the rendered 
rendering control signals based on a current location within scene> aQd the ^nteM programming in general. To that end, 
the scene graph. The compositor/renderer 180 outputs the the MPEG-J systems and methods of this invention define 
rendered scene to a downstream processing subsystem, such application program interfaces (APIs) to the various com- 
as a display subsystem that displays the rendered scene on ^ poa ents of the MPEG-4 browser or presentation engine 100. 
a display device. The compositor/renderer 180 may respond Further, the MPEG-J systems and methods of this invention 
to very basic user interactions, such as mouse clicks, etc. describes the delivery of application programs using the 

In one exemplary embodiment of the MPEG-4 browser standard MPEG-4 elementary streams delivery mechanism. 
100 shown in FIG. 1, MPEG-4 coded data read from a Besides delivering application programs to the MPEG-4 
storage device or received over a distributed network, such 20 browser or presentation engine, the MPEG-4 browser or 
as a LAN, a WAN, the Internet or an intranet, goes through presentation engine can be configured with local application 
the delivery multimedia integration framework subsystem programs that use the same API's. Examples of sources of 
120 and the demultiplexer layer 100. In the demultiplexer such local programs are programs that are downloaded, as 
layer 100, FlexMux-PDU's pass through a Sync Layer well as commercially available programs that enhance the 
resulting in unformatted SL-PDUs of each media type, such 25 performance and functionality of the MPEG-4 browser or 
as, for example, encoded audio streams, encoded video presentation engine. It should be appreciated that local 
streams, encoded speech streams or encoded facial anima- programs will generally have no knowledge of the delivered 
tion streams. The unformatted SL-PDUs of each media type specific MPEG-4 content. In contrast, the delivered pro- 
are then buffered in the respective decoder buffers 130 and grams that originate from the same author as the audio/ 
are offered to the corresponding media decoders 150. Also, visual MPEG-4 content will generally have knowledge of 
any SL-PDUs corresponding to scene description represen- the delivered specific MPEG-4 content, 
tation are input to the binary format for scene decoder 140. The architecture of the MPEG-J application engine 200 
The output of the binary format for scene decoder 140 goes ta k es i nt0 consideration the resources available on the 
to the scene graph controller 160. The output of the media ^ underlying multimedia data processing system 10. The a 
decoders 150 as well as the scene graph controller 160 feeds architecture of the MPEG-J run time environment 20 iso- 
the compositor/renderer 180. The output of the compositor/ lates distinct components of the MPEG-J application engine 
Tenderer 180 is the scene for presentation. 200, the design of application programming interface that 

The MPEG-J systems and methods according to this reflects these components, and the characterization of inter- 
invention are flexible programmatic control systems and 4Q actions between these components. 

methods that represent an audio -visual session in a manner The distinct components of the MPEG-J run time envi- 

that allows the session to adapt to the operating character- ronment 20 include execution and presentation resources, 

istics of the multimedia data processing system on which the decoders, network resources, and the scene graph. The 

MPEG-J application, the MPEG-4 browser and an execution architecture of the MPEG-J application engine 200 assumes 

engine, such as a display subsystem, are executing. The 45 that decoding and presenting resources of the underlying 

MPEG-J systems and methods according to this invention multimedia data processing system 10 are limited. The 

allow graceful degradation under limited or time varying execution and presentation resources components abstract 

resources. The MPEG-J systems and methods according to access to information on such static and dynamic resources 

this invention also provide the ability to respond to user in the MPEG-4 browser or presentation engine and notifi- 

interaction and provide enhanced multimedia functional- 5Q cation during changes in such resources. The execution and 

ities. presentation resources components provides for some mini- 

The MPEG-J systems according to this invention are mal control of the static and dynamic resources in the 

programmatic systems that specify interfaces for inter- MPEG-4 browser or presentation engine, 

operation of an MPEG-4 media browser with Java code. By The decoders component abstracts the media decoders 

combining MPEG-4 media and safe executable code, con- 5S used to decode the received media streams. The decoders 

tent creators may embed complex control mechanisms with component also abstracts the programmatic control of the 

their media data to intelligently manage the operation of the media decoders and the manipulation of the media decoders 

audio-visual session. In the MPEG-J systems according to that is used to add extra functionality to the MPEG-4 

this invention, Java code is delivered as a separate elemen- browser or presentation engine. 

tary stream to the multimedia data processing system. The 60 As described above, the MPEG-4 browser or presentation 
Java code is directed to a MPEG-J run time environment of engine receives media streams. The network resources co ra- 
the multimedia data processing system. The MPEG-J run ponent abstracts the control of the received media streams, 
time environment includes a Java Virtual Machine, which The network resources component also abstracts the media 
provides the Java code access to the various components of pipeline or pipelines that transport and present the media 
the MPEG-4 browser 100. 55 streams to the MPEG-4 browser or presentation engine. 

FIG. 2 is a functional block diagram of one exemplary As described above, each session presented by the 

embodiment of the components of the MPEG-J operating MPEG-4 browser or presentation engine has a Scene Graph 
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to represent and spatially and temporally locate the audio- tion framework subsystem 120 using the interface channels 
visual objects of that session. The scene graph component 212 and 221 of the network application programming inter- 
abstracts access to and control of the scene graph. face Nw API. The scene graph manager 250 interfaces the 
FIG. 3 shows a functional block diagram illustrating in MPEG-J application controller 220 with the scene graph 
greater detail a first exemplary embodiment of the MPEG-J s controller 160 using the interface channels 222 and 252 of 
application engine 200 of FIG. 2 according to this invention. the scene graph application programming interface SG API. 
The MPEG-J systems and methods according to this inven- The resource manager interfaces the MPEG-J application 
tion allows a combination of MPEG-4 media and safe controller 220 with the scene graph controller 160, the 
executable code, so that the content creators can embed decoder buffers 130, the media decoders 150, the compositor 
complex control mechanisms within their media data to 1Q buffers 170 and the compositor/renderer 180 using the 
intelligently manage the operation of the audio -visual ses- interface channels 261-265 of the resource manager appli- 
sion. In the first exemplary embodiment shown in FIG. 3, the cation programming interface RM API. Hie interface chan- 
architecture of the MPEG-J systems according to this inven- nel of the device application programming interface Dv API 
tion includes, in the lower portion of the MPEG-J run time allows users to directly interact with and control the 
environment 20, the exemplary embodiment of the MPEG-4 J5 MPEG-J application controller 220 using one or more of the 
browser or presentation engine 100 shown in FIG. 2. The I/O devices 280. Thus, the user input, through the MPEG-J 
MPEG-J system or application engine 200 forming the application controller 220, can be used to control the various 
upper portion of the MPEG-J run time environment 20 elements of the MPEG-4 browser or presentation engine 
specifies the interfaces and control structures that control the 100. 

MPEG-4 browser or presentation engine 100. The scene graph manager 250 allows direct control of the 

As shown in FIG. 3, the first exemplary embodiment of binary format for scene graph that specifies the scene to be 

the MPEG-J system or application engine 200 includes a rendered. In particular, the scene graph manager 250 is able 

network manager 210, the MPEG-J application controller to turn off portions of the scene graph to better match the 

220, a buffer 230, a class loader 240, a scene graph manager processing and/or memory resources of the underlying mul- 

250, a resource manager 260, and one or more input/output 25 ti media data processing system 10 available to the MPEG-J 

(I/O) devices 280. The I/O devices 280 can be any known or run time environment 20 to the resources needed to render 

later developed input or output device, including a mouse, a the scene. Thus, the scene graph manager 250 controls the 

keyboard, a touchscreen, a pen device, a set-top remote scene graph controller 160 to modify the scene graph or the 

control device, a television remote control device, a display generated multimedia scene in view of the current system 

device, such as a television, a CRT- type display, an LCD- 30 resources, user inputs, and/or any embedded control 

type display, an LED-type display, a printer, or the like. mechanisms, such as behavior control commands. In 

The network manager 210 is connected to the delivery particular, the scene graph or the generated multimedia 

multimedia integration framework subsystem 120 and the scene is modified to gracefully or controllably degrade the 

MPEG-J application controller 220 through interface chan- generated multimedia scene. 

nels 212 and 221, respectively, of a network application 3S The resource manager 260 manages system resources of 

programming interface Nw API. The buffer 230 is connected the underlying multimedia data processing system 10, 

by a data channel 116 to the demultiplexer layer 110 and a allowing the regulation of overall performance. This 

data channel 232 to the class loader 240. The class loader includes controlling the scene graph controller 160, the 

240 is connected to the MPEG-J application controller 220 media buffers 130, the media decoders 150, the compositor 

by a data channel 242. The scene graph manager 250 is 4Q buffers 170, the compositor/renderer 180 and MPEG-J run 

connected to the scene graph controller 160 and the MPEG-J time environment 20 functionalities and resources in view of 

application controller 220 through interface channels 252 the current system resources, user inputs, and/or any embed- 

and 222, respectively, of a scene graph application program- ded control mechanisms, such as behavior control com- 

ming interface SG API. mands. In particular, the resource manager 260 controls 

The resource manager 260 is connected to the MPEG-J 45 these elements to gracefully or controllably degrade the 

application controller 220 through interface channels 223, generated multimedia scene. 

224 and 225 of a functionality application programming The network manager 210 provides a path for the 
interface Fn API, a media decoders application program- MPEG-J application controller 220 to send requests or 
ming interface MD API, and a resource manager application receive data through the MPEG-J application controller 220. 
programming interface RM API, respectively. The resource 50 The network manager 210 can also monitor the network 
manager 260 is also connected through interface channels resources of the distributed network, and can control the 
261-265 of the resource manager application programming connection to the distributed network in view of the current 
interface RM API to the scene graph manager 160, the network resources, user inputs, and/or any embedded control 
decoder buffers 130, the media decoders 150, the compositor mechanisms, such as behavior control commands. The net- 
buffers 170 and the compositor/renderer 180, respectively. 55 work manager 210 also allows the MPEG-J application 
The I/O devices 280 are connected to the MPEG-J applica- controller 220 access to the network resources of the dis- 
tion controller 220 through an interface channel 226 of a tributed network. The MPEG-J application controller 220 
device application programming interface Dv API. The can then control the MPEG-4 browser or presentation engine 
network application programming interface Nw API, the 100 in view of the current network resources, user inputs, 
scene graph application programming interface SG API, the eo and/or any embedded control mechanisms, such as behavior 
functionality application programming interface Fn API, the control commands. 

media decoders application programming interface MD API, FIG. 4 shows a functional block diagram illustrating in 

the resource manager application programming interface greater detail a second exemplary embodiment of the 

RM API, and the device application programming interface MPEG-J application engine 300 of FIG. 2 according to this 

Dv API are described in greater detail below, 65 invention. The second exemplary embodiment of the 

The network manager 210 interfaces the MPEG-J appli- MPEG-J application engine 300 is generally identical to the 

cation controller 220 with the delivery multimedia integra- first exemplary embodiment of the MPEG-J application 
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engine 200 shown in FIG. 3, except that the second exem- interface channel 426 of the device application program- 

plary embodiment of the MPEG- J application engine 300 ming interface Dv API. 

separates the control and data channels and reorganizes the Likewise, the scene graph manager 450 is no longer 

interface and control channels between a number of the connected to the scene graph controller 160 over the inter- 

J?SK!2 n f nts ? f ^ e SCCOnd C ^ P T lai T embodimcnt of 1 the 5 face channel 452 of the scene graph-external application 

MPEG-J application engme 300. n the second exemplary interface lication prograrnrrjing interface SG-EAI API. 

embodiment of the MPEG-J application engine 300, the M [n ^ secQnd k embodiment of the MPEG-J 

network manager 210 has been deleted, with the network ... . - rtA r , . A . 

. . , i. i mi i j j * *i * .u a v application engine 300 shown m FIG. 4, the scene graph 

interface channel 321 connected directly to the delivery vr „ + A . ,• 6 L 

u - j- • t e „ i . 4 « ha tu controller 160 is connected to the application manager 4v0 

multimedia integration framework subsystem 120. rhe „ rt . . - . , An + r X. .... to 

*AT*m i i- 7- * ii HA i j i- „• 10 over the interface channel 491 of the application manager 

MPEG-J application controller 320 includes an application ... . i AW \ r nT „ f, 

mana er 390 application programmmg interface AM API. However, the 

* . * t-™ * * i ^„ • scene graph manager 450 is now connected to a switch 456 

As shown in FIG. 4, the scene graph manager 350 is stil ^ a data/eontrol channe i 452 of ^ scen6 gr aph-external 

connected to the scene graph controller 160 and the MPEG J hcitioa iDterface a p pl i calitm programming interface 

application controller 320. However, the scene graph man- 15 §q.£^j ^pj 

aeer 350 is now connected to the scene graph controller 160 ™ * . * . . . , . , 

* 1 u i id „„j w ttcr ! The A channel out of the switch 456 directs the control 

over a control channel 352 and to the MPEG-J application , A . _ iL , , 

controller 320 over an interface channel 322 of a scene ™ d/oT data to" 11 ^ f en / e f ^ mana 8 e f r 450 l ° a 

graph-external application interface application program- bmar y fo ™ at f ™ : " \ ^ 

ming interface SG-EAI API. The scene graph controller 160 M s <* ne encoder 454 ln P uts data , 411(1 control signals from 

is now connected to the application manager 390 over an ^ scene graph manager 450 and converts them into new 

interface channel 391 of an application manager application ^ co 1 ntent ' ^ bma 7 f ° r , sce ° e e f ° d £. «** 

„ n . i„ t „ fnM A \/r a dt outputs the new scene content over a data channel 457 to the 

programming interface AM API. * j j l *u 

«... i , . ^ bmary format for scene decoder 140, where the new scene 

Similarly, the resource manager 360 remains connected to . . ■ j j j ■ -i- *uui c ~ • * i 

41 _ t i- * ii nn .u u • . content is decoded mdistinguishably from ongmal scene 

the MPEG-J application controller 320 through interface ?< t . . . . , u °, . ,/ , ^ . ° _ 

u i M1 j p .« £• r# /3 content and output over the data channel 142 to the scene 

channels 323 and 325 of the functionality application pro- , . « *n 

■ * r r Am ,■ graph controller 1 60. 

gramming interface Fn API and the resource manager apph- S. n . , * . -.4,^,.. 

caUon programming interface RM API, respectively. ]? e * channe °" 1 of ' he SWItch 45 u 6 th / e ? )ntr ° 1 

However, the resource manager 360 is now connected to the and/or d »ta signals from the scene graph manager 450 to the 

application manager 390 over the interface channel 324 of 30 sc f « 8 ra P h c<""roller 160 over a data channel 458. Thus, 

the media decoders application programming interface MD whlle the s f ne mana g« 450 contmuc 1 s 10 contro1 ' h « 

API. Furthermore, the resource manager 260 is now con- scene g ra P h controller 160, the scene graph manager 450 

nected through control channels 361-365 to the scene graph not « teecfl y mterface mth thc sccnc W b controller 
manager 160, the decoder buffers 130, the media decoders 

150, the compositor buffers 170 and the compositor/re nderer 35 However, it should be understood that the binary format 

180, respectively. f° r scene encoder 454 does not have to be implemented in 

The network application programming interface Nw pro- the third exemplary embodiment of the MPEG-J application 

vides a path for the MPEG-J application controller 320 to cn g ine 400 - Rather > as in the first and sccond c «°f ^ 

send requests or receive data through the MPEG-J applica- embodiments of the MPEG-J application engine 200 and 

tion controller 320. The MPEG-J application controller 320 4 o 300 > the sceQe £ ra P h manager 450 can be directly connected 

can also monitor the network resources of the distributed t0 the S ra P h controller 160 over the interface channel 

network, and can control the connection to the distributed 452 of the scene S ra P h application programming interface 

network in view of the current network resources, user SG-EAI API. 

inputs, and/or any embedded control mechanisms, such as The interactivity manager application programming inter- 
behavior control commands. The network application pro- 45 face 1M ^ l aad the interactivity manager 470 permit a 
gramming interface Nw also allows the MPEG-J application greater degree of user input and interactivity with the 
controller 320 access to the network resources of the dis- multimedia scene generated by the MPEG-4 browser or 
tributed network. The MPEG-J application controller 320 presentation engine 100. Thus, the user input, through the 
can then control the MPEG-4 browser or presentation engine MPEG-J application controller 220, can be used to control 
100 in view of the current network resources, user inputs, 50 and interact with the various elements of the MPEG-4 
and/or any embedded control mechanisms, such as behavior browser or presentation engine 100 to a greater extent, 
control commands. The MPEG-J application engines 200-400 outlined above 
FIG. 5 shows a functional block diagram illustrating in < are specific exemplary embodiments of a multimedia brows- 
greater detail a third exemplary embodiment of the MPEG-J ing control subsystem of a multimedia browsing system 
application engine 400 of FIG. 2 according to this invention. 55 according to this invention. The MPEG-4 browser or pre- 
The third exemplary embodiment of the MPEG-J applica- sentation engine 100 outlined above is a specific exemplary 
tion engine 400 is generally identical to the second exem- embodiment of a multimedia browsing subsystem of a 
plary embodiment of the MPEG-J application engine 300 multimedia browsing system according to this invention, 
shown in FIG. 4, except that an interactivity manager 470 is The multimedia browsing subsystem generates a multimedia 
positioned between the MPEG-J application controller 420 60 scene based on a plurality of data streams. The data streams 
and the I/O devices 480, and the scene graph manager 450 include data defining a number of multimedia objects and 
is not directly connected to the scene graph controller 160. data defining control programs or mechanisms that are 
As shown in FIG. 5, the MPEG-J application controller 420 associated with the multimedia objects, 
is connected to the interactivity manager 470 over an The multimedia browsing control subsystem (e.g., the 
interface channel 427 of an interactivity manager application 65 MPEG-J application engines 200-400) according to this 
programming interface IM API, while the interactivity man- invention controllably degrades (i.e., graceful degradation 
ager 470 is connected to the I/O devices 480 over the device of) the multimedia scene generated by the multimedia 



07/25/2004, EAST Version: 1.4.1 



US 6,631,403 Bl 

13 14 

browsing subsystem (e.g., the MPEG-4 browser or presen- . engine 100 in the form of an MPEG-4 elementary stream, 

tation engine 100) in the face of limited system resources This implies that the "MPEG-J elementary stream" has an 

available to the multimedia browsing subsystem, and/or in associated Elementary Stream Descriptor. Further, a stream 

view of complex control mechanisms, or behavior control « M PEG-J Stream" is defined in the StreamType table 

commands, embedded within the data defining the multi- s if . TC _ 1yl/in - , ™ t a * ij u 

media scene for intelligently managing the generation of the ( Table 9 ) in ISO/IEC 14496-1. The MPEG-J data could be 

multimedia scene. The multimedia browsing control sub- classes or serialized objects transmitted as byte code from 

system according to this invention separately enables a servers to clients. Serialized objects are expected to accom- 

mimber of enhanced real-time and other functions for con- pany classes that have knowledge about handling those 

trollably degrading and interacting with the multimedia objects, 
browsing subsystem in response to user inputs. 

It should be understood that each of the buffers, managers, » , . . x*™^ a 

controllers, decoders, application programming interfaces An MPEG-J session is initiated when the MPEG-4 

and other elements of the multimedia browsing systems browser or presentation engine 100 receives an MPEG-J 

shown in each of FIGS, 1-5 can be implemented as portions ]5 object descriptor. In response to receiving the MPEG-J 

of a suitably programmed general purpose computer used to object descriptor, the MPEG-4 browser or presentation 

implement the multimedia data processing system 10. engine 100 opens an MPEG-J elementary stream via a 

Alternatively, each of the elements shown in each of FIGS. DMIF channel in the demultiplexer layer 110. The MPEG-J 

1-5 can be implemented as physically distinct hardware elementary stream is an SL packetized stream, similar to all 

circuits within an ASIC, or using a FPGA, a PDL, a PLAor M other MPEG-4 streams. The MPEG-4 browser or presenta- 

a PAL, or using discrete logic elements or discrete circuit tion engine 100 then delivers the& access units of the 

elements, depending on the particular hardware used to MPEG-J elementary stream to the buffer 230 of the class 

implement the multimedia data processing system 10. The i oa( j er 240 over the data channel 116. The class loader 240 

particular form each of the circuits shown in each of FIGS, i oa ds the classes. 

1-5 will take is a design choice and will be obvious and 2j 
predicable to those skilled in the art. For example, the 

MPEG-J application engine and the MPEG-4 presentation An MPEG-J application that uses the MPEG-J application 

engine can each be implemented as software executing on a programming interfaces of this invention to control the 

programmed general purpose computer, a special purpose MPEG-4 browser or presentation engine 100 can either be 

computer, a microprocessor or the like that is used to 3Q local or remote relative to the underlying multimedia data 

implement the multimedia data processing system 10. In this processing system 10. In the case of a remote application 

case, the MPEG-J application engine and the MPEG-4 that is received in the MPEG-J elementary stream, the 

browser can each be implemented as an embedded routine, remote application must implement a remote application 

as a resource residing on a server, or the like. interface. The header to the remote application and the 

The channel 102 and the back channel 104 can be 35 remote application interface can be implemented using any 

implemented using any known structure or apparatus for appropriate and/or relevant techniques and programming 

transmitting the MPEG -encoded data to the MPEG-4 structures known to those skilled in the art. 
browser. Thus, the channel 102 and the back channel 104 

can be implemented using a public switched telephone , , , , . . 4 . 

network, a local or wide area network, an intranet, the 40 . MPEG-J scene graph capabilities are always based on the 

Internet, a wireless transmission channel, any other distrib- *&*y integrated or content provider oriented model. In 

uted network, or the like. general, the first stream received is the master* scene 

There are two scenarios in which MPEG-J may be used. 6 ra P h - Subsequent streams are added as "subtrees" at des- 

In one possible scenario, the content provider designs all ig nated nodes ™ the master ^ S ra P n ' ^ master ^ scene 

MPEG-J content, and all of the necessary or desirable 45 graph must acquire its subtrees. However, any tree can be the 

add-ons for local use and interactivity, such as including master scene graph. Nonetheless, subtrees inherit acquisi- 

commercials, local programming information, scene tion limits, 
interactivity, and other subsequent modifications. Hiis is 

desirable for content providers, at it allows incremental ^ MPEG .j application programming interface is not a 

updates. That is, change of state can be done via binary 50 s i ng i e application programming interface but a coUection of 

format for scene updates. In a second possible scenario, the application progr amming interfaces that address various 

client dynamically controls the scene. This is desirable for ^educs for a flejd51e MPEG _ 4 system . Java language is 

set-top manufacturers. It should also be appreciated that . 4 . . . * M 

, F . . « . . , l ^ j u used for application programming interface specification. 

local programming and interactivity can also be provided by wnr-£ T r *• ■ • i a ~.u~~ ~f 

• ; j- u ♦ *u • * i * * -a a The MPEG-J application engine includes a number of 

an intermediary between the original content provider and 55 . - , . . ■ ■ * * 

the client or categories of application programming interlaces, as out- 

HoweVer, it should be appreciated that changes in non ^ned above. Tnese categories of application programming 
updatable nodes may not be possible. The MPEG-J systems interfaces include the application manager application pro- 
and methods of this invention may eventually need to serve gramming interface; the scene graph manager apphcation 
both of these scenarios. Since the first scenario is more 60 Programming interface; the resource manager apphcation 
deterministic than the second scenario, the exemplary programming interface; the interactivity manager applica- 
embodiments of the MPEG-J systems and methods of this tion programming interface; the media decoder apphcation 
invention that are described herein mainly support the first programming interface; the functionality application pro- 
scenario while including the hooks to partially support the gramming interface; the networking application program- 
second scenario. 65 naing interface; and the device application programming 

Application programs based on the MPEG-J API's are interface. Table 1 outlines these application programming 

made available to the MPEG-4 browser or presentation interfaces and their main classes or interfaces: 
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Explanation 

Direct interface to scene graph to deal with 
properties 

Means by which MPEG-J apps access and 
manipulate the BIFS player 
Centralized facility for managing system 
resources 

Centralized facility for managing user 
interactivity with scene 
Access and control to the decoders used to 
decode the audio-visual objects. 
Decoding of video objects 
Decoding of image texture objects 
Decoding of wireframe mesh objects 
Decoding of Face objects 
Decoding of speech and Audio objects 
Decoding of TTS objects 
Decoding of structured audio objects 
Progressive decoding of image texture 
object under user control 
Selective decoding of active objects for 
enhancement under user control 
Selective decoding of object based on 
viewpoint under user control 
Selective decoding of full or partial objects 
based on transparency and user control 
Selective decoding of object for playback 
in trick mode (FF,FR) under user control 
Selective decoding and enhancement of 
audio object under user control 
Access and control of the network 
components of the MPEG-4 browser. 
Access and control of back channel 
An interface for input device drivers 
An interface for user input device drivers 
An interface for audio devices 
An interface for smart card devices 



List of MPEG-J Application Programming Interfaces 



API Category 


Main Classes Or 
Interfaces 


Application Manager 


Application Manager 


Scene Graph 


SBrowser 


Manager 




Resource Manager 


Resource Manager 


Interactivity Manager 


InteractivityManager 


Media Decoder 


MPDecoder 




Video Decoder 




TextureDecoder 




Mesh Decoder 




FaceDecoder 




AudioDecoder 




TTSDecoder 




StructAud ioDecoder 


Functionality 


Pro gressivelmage 




HotObject 




DirecDecoding 




Transparency 




TrickModc 




Adjust Audio 


Network 


NctworkManager 




BackChannel 


Device . 


Input Device 




UserlnputDevice 




AudioDevice 




SmartCard 



Packages are a means to organize the implementations of 
application programming interfaces. For the list of MPEG-J 40 
application programming interfaces outlined above for the 
systems and methods of this invention, one exemplary set of 
packages includes an "mpgj.sys" package, an "mpgj.dec" 
package, an "mpgj.func" package, and an "mpgj.dev" pack- 
age. The "mpgj.sys" package contains classes for 45 
application, scene graph, resource and interactivity manager. 
The "mpgj.dec" package contains classes for Media decod- 
ing. The "mpgj.func" package contains classes for function- 
alities including interaction. The "mpgj.dev" package con- 
tains classes that provide interfaces to various input, output, 50 
sound and video devices. 

A local or remote MPEG application will use well-defined 
sets of MPEG-J application programming interfaces to inter- 
act with the underlying MPEG-4 browser or presentation 
engine 100. These Java™ application programming inter- 55 
faces are implemented on all compliant MPEG-4 browsers 
or presentation engines. The details of the required imple- 
mentation of these application programming interfaces can' 
be found in the MPEG-4 systems Version 1 Standard (ISO/ 
IEC 14496-1). It is expected that the local or remote 60 
MPEG-J application engine can call methods defined in the 
MPEG-J application programming interfaces and those from 
a minimal set of Java™ packages. The MPEG-J application 
programming interfaces are functionally divided into a net- 
work (Nw) application programming interface' category; a 65 
resource manager (RM) application programming interface 
category; a media decoder (MD) application programming 



interface category; a scene graph (SG) application program- 
ming interface category; and a functionality (Fn) application 
programming interface category. 

The resource manager (RM) application programming 
interfaces help to manage resources on the underlying 
multimedia data processing system 10 during an MPEG-J 
session. These resource manager (RM) application program- 
ming interfaces can be used to adapt an MPEG session to a 
given data processing system. The main components of the 
resource manager (RM) application programming interfaces 
are the resource manager interface and the event model 
defined by the resource manager. 

The resource manager interface supports a number of 
different functionalities, including access to the media 
decoder associated with a particular node, providing a list of 
available media decoders of a specific type, access to the 
priority associated with a media decoder, event models with 
downloadable event handlers, events mechanisms for stream 
and decoder problems, and events mechanisms for Tenderer 
problems. That is, the resource manager interface provides 
access to the renderer, the media decoders and their priori- 
ties. The resource manager interface also defines methods to 
gain access to a media decoder given a node in the scene 
graph and change it to another available media decoder. The 
media decoders and the decoder functionality application 
programming interfaces rely on the resource manager to 
obtain the instance of a media decoder. 

The resource manager interface provides a centralized 
facility for managing resources. The resource manager inter- 
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face is a collection of a number of classes and interfaces, ager. The resource manager also provides the instance of the 

including MPDe co de rEventGenera tor, decoder associated with a node in the scene graph. 

MPRendererEventGenerator, MPDecoderMediaListener, The media decoder (MD) application programming inter- 

and MPRendererMediaListener interfaces, and faces support a number of different functionalities, including 

"MPDecoderMediaEvents", "ResourceManager", 5 starting, stopping, pausing, and resuming a decoder, attach- 

"Renderer", and "MPRendererMediaEvents" classes. ing and detaching a decoder to elementary streams, and 

For each media decoder, the resource manager (RM) getting the type and other attributes of a decoder, 

application programming interface has an installation of a The MPDecoder is an interface that abstracts the most 

class that implements MPDecoder or a sub -interface. These generic decoder. The MPDecoder interface facilitates attach- 

media decoder installations generate the events for different 30 ing and detaching decoders to elementary streams. It also 

situations. The resource manager (RM) application pro- provides for access to the type, vendor, and an instance 

gramming interface can handle events if necessary in addi- number of the decoder. A number of interfaces that abstract 

tion to the event handlers in the application. The MPEG-J decoders, like a scalable video decoder, i.e., the SclVideo- 

application engine can receive the Event handlers as byte- Decoder interface, a structured audio decoder, i.e., the 

code in the bit stream. 15 SclAudio Decoder interface, that are sub-interfaces of 

Apart from implicitly specifying this event model, the MPDecoder interface are also denned. These subinterfaces 

resource manager (RM) application programming interface provide access and control to specialized features of that 

also provides access to the media decoders and their priori- particular decoder. 

ties. Given a node in the scene graph, resource manager The scene graph (SG) application programming interfaces 

(RM) application programming interface provides access to 20 provide mechanisms for controlling the scene graph and/or 

the media decoder associated with that node. The resource the scene graph controller. The scene graph (SG) application 

manager (RM) application programming interface also programming interface allows the behavior of a node of the 

facilitates setting and getting the media decoder priorities. scene graph to be changed and allows the scene graph to be 

The resource manager (RM) application programming inter- edited. TTie scene graph (SG) application programming 

face also enables changing which media decoder is associ- 25 interface supports a Browser Object. The scene graph (SG) 

ated with a particular node. application programming interface also supports updatable 

The resource manager (RM) application programming nodes and/or MPEG-J accessible nodes of the scene graph, 

interfaces also define decoder and renderer events. The For the content provider scenario described above, the 

corresponding interfaces for event generators and listeners, 3Q scene graph (SG) application programming interface sup- 

such as event handlers, are also defined. The resource ports user priority only on those objects that the content 

manager (RM) application programming interfaces implic- provider has indicated as allowable, and scene graph 

itly define an event model for graceful degradation. For each manipulation only corresponding to those objects that the 

media decoder, the resource manager (RM) application content provider has indicated as allowable. For the user 

programming interfaces would have an instantiation of a 35 scenario described above, the scene graph (SG) application 

class that implements an MPDecoder interface or a sub- programming interface supports user priority and scene 

interface. These decoder instances generate the events cor- graph manipulation on all objects. One exemplary embodi- 

responding to different situations. These events can handle ment of the scene graph (SG) application programming 

events if necessary in addition to the default event handlers interface allows for tagging multiple streams as sources and 

in the application. The MPEG-J application engine can 4Q selecting one stream as the master. 

receive the event handlers as bytecode in the bitstream. [ n the scene graph (SG) application programming inter- 
Program execution may be contingent upon the configu- face according to this invention, there is a browser for each 
ration of the particular data processing system and its subtree, and is the interface between MPEG-J and that 
capabilities. An MPEG-J session may need to be aware of its subtree. There are two modes for updating a node, a binary 
environment, so that it is able to adapt its own execution, and 45 format of scene stream mode and a direct node access mode, 
the execution of the various components as they may be The scene graph (SG) application programming interface 
configured and running in the MPEG-4 browser 100. according to this invention supports binary format for scene 
The network (Nw) application programming interfaces updates, turning off the rendering of certain objects, attach- 
allow control of the network component of the MPEG-4 ing objects to nodes, and the like. 

browser or presentation engine 100. Through the network 50 The functionality (Fn) application programming inter- 

(Nw) application programming interfaces, the MPEG-J faces provide mechanisms for controlling the media decod- 

application engine can interact with the distributed network ing process. In particular, a number of useful predefined 

to which the MPEG-4 browser or presentation engine 100 is functionalities become possible under user control. The 

connected. The network (Nw) application programming functionality (Fn) application programming interfaces are 

interfaces allow the MPEG-J application engine to monitor 55 dependent on other parts of the other parts of the MPEG-J 

the network resources and controllably degrade the multi- application engine 200, such as the media decoder (MD) 

media scene in view of the network resources. It should be application programming interfaces, the resource manager 

appreciated that those skilled in the art would be able to (RM) application programming interfaces, and the scene 

provide network (Nw) application programming interfaces graph (SG or SG-EAI) application programming interfaces, 

for the MPEG-4 browser or presentation engine 100 and the 60 Accordingly, the functionality (Fn) application program- 

MPEG-J application engine described herein using known ming interfaces will most likely change as these other 

techniques and programming structures. application programming interfaces mature. 

The media decoder (MD) application programming inter- The current implementation of the functionality (Fn) 

faces facilitate basic control, such as start, stop, pause, application programming interfaces support a number of 

resume and the like, of all the installed decoders in an MPEG 65 different functionalities, including a Progressive! mage func- 

session. A list of decoders that are currently installed and tion; a HotObject function; a DirecDecoding function; a 

those available can be obtained through the resource man- Transparency function; and a TrickDecoding function. 
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The functionality (Fn) application programming inter- tifies itself with nodeld, while the string specifies the prop- 
faces include interfaces for each of these functionalities. The erty that it is interested in. 
Progressivelmage interface triggers the action of progressive Class mpgj.sys.NodeCanCallAM 
refinement of quality of the image being decoded under user 

control. The Progressivelmage interface extends the 5 java.lang.Object 

"SclTextureDecoder", which is used to decode image tex- | 
ture streams. The HotObject interface triggers the action of 
enhancement of a visual object provided that the object is a 
hot object. Thus hot objects have some enhancement streams 

associated with them that are triggered when needed. The 10 + ---mpgj.sys.NodeCanCallAM 
HotObject interface extends the "SclVideoDecodcr", which 
is used to decode base and enhancement streams. 

The DirecDecoding interface allows creation of visual public abstract class NodeCanCallAM 

objects that are directionally sensitive. The DirecDecoding extends vrml.external.Node 

interface is most easily explained by assuming a bitstream 15 15 a special node object that is capable of calling 

composed of a number of static video object planes (visual Application Manager. 

Ops) coded as an audio-video object such that depending on Constructors 

the user interaction, video object planes corresponding to public NodeCanCallAM ( ) 

one or more viewpoint are decoded as needed. The Trans- Methods 

parency interface allows formation of visual objects with 20 ^ rcgislcrAM(App]icationManagcr am) 

transparency information. The Transparency interface also ^ [& iQ be called by MPEG .j application. 

supports selective decoding of objects under user control. The MPEG .j Sccne Graph application programming 

The user may not need to decode all objects or portions of specification is set forth below: 

object due to bandwidth/computing resources. If certain ^ In(erface rapgj . sys .ScenegrapbManager 

objects or portions of object are hidden they may not need 25 

to be decoded. The portions of objects not to be shown can P ubllc interface ScenegraphManager 

be marked with the transparency color, which can be Interface that MPEG-J applications access and mampu- 

extracted to form transparency information. This process late the BIFS P laver - 

can also be used to highlight or mark objects. The Trick- Methods 

Decoding interface allows selective decoding of audiovisual 30 public void createNode(java.lang.String nodeName, 

objects for playback in trick mode (i.e., fast forward (FF) or java.lang.String[ ] fields, 

rewind (FR)) under user control. java.lang.Stringf ] values) 

Although the MPEG-J systems, methods and application Creates a node with nodeName, fields, and field values, 

programming interfaces of this invention do not cover a is a one to one correspondence between the field 

capability manager application programming interface, a 35 namcs and ^ values tnc field namcs and thc valucs arc 

profile application programming interface, a terminal capa- g iven in the same order - [f a field * ™ssing> the default 

bilities application programming interface, a section filter valuc 15 uscd as specified in the systems document. The 

application programming interface, a service information/ underlying implementation is expected the create a node 

object content information application programming exact] y lhe same wa y m which lt 1S done in the case of 

interface, or a timing application programming interface, 40 creating a node in the BIFS stream, 

these application programming interfaces are fully described Parameters: 

in the MPEG-J standard, and are specifically described in nodeName — The name of the node 

"Text for VM 6.0 Systems Version 2 (MPEG-J)", ISO/1EC fields— field names 

JTa/SC29/WGll/W2741p3, Seoul, Korea, Mar. 1999. The values— values of fields 

network application programming interface is described in 45 public void addRoute(vrml.external.Node fromNode, 

greater detail in "Text for ISO/IEC 14496-1/PDAMI java.lang.String eventOut, 

(MPEG-4 version 2 MPEG-J)", ISO/IEC JTCI/SC29/ vrml.external.Node toNode, 

WG11/N2739p3, Seoul, Korea, Mar. 1999. java.lang.String eventin). 

The MPEG-J application programming interface specifi- 5Q Adds a route between two nodes, from an eventOut to an 

cations for the various MPEG-J application programming eventin. If the ROUTE already exists, this method silently 

interface described above are set forth below, exits. It does not attempt to add a second parallel ROUTE. 

The MPEG-J Application Manager application program- public void deleteRoute(vrml.external.Node fromNode, 

ming interface specification is set forth below: java.lang.String eventOut, 

Interface mpgj.sys.ApplicationManager public interface 5S vrml.external.Node toNode, 

ApplicationManagcr java.lang.String eventin) 

This interface allows the scene graph to access informa- Deletes a route between two nodes. If the route does not 

tion from application. This interface is implemented by the exist, the method silently exits, 

application, the application manager is registered with a public void beginupdate ( ) 

node that can call the application. 60 me output from the external interface to the scene 

Constructors graph as the code is about to begin a series of updates. No 

public ApplicationManager ( ) events will be passed to the scene graph. They will be 

Methods buffered pending release due to a subsequent call to end- 
Public java.lang.String getproperty(int nodeld, java- Update. This call is a nesting call which means subsequent 
.lang.String property) 65 calls to beginupdate are kept on a stack. No events will be 
This method will be called by node object to get property released to the scene graph until as many endUpdates have 

information from the application or applet. The node iden- been called as beginUpdate. 
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public void endupdate ( ) 

Releases the output of events from the external interface 
into the scene graph. All events posted to this point from the 
last time that begimipdate was called are released into the 
scene graph for processing at the next available opportunity. 
This call is a nesting call which means subsequent calls to 
beginupdate are kept on a stack. No events will be released 
to the scene graph until as many end Updates have been 
called as beginupdate. If no beginUpdate has been called 
before calling this method, it has no effect. 

public vrml. external. Node getNode(java.lang.String 
name) 

Gets a DEF node by name. Nodes given DEF names in the 
root scene graph are available to be retrieved by this method. 
DEFed nodes in Mines are not available. 

Although they have not been explicitly included here, all 
the EAI API's except the Browser interfaces and classes of 
the Virtual Reality Markup Language can be used for scene 
graph control. 

The MPEG- J Resource Manager application program- 
ming interface specification is set forth below: 
Interface mpgj.sys.MPDecoderEventGenerator 
Subinterfaces: MPDecoder, ScdDecoder 
public interface MPDecoderEventGenerator 
Methods 

public void addMPDecoderMediaListener 

(MPDecoderMediaListener 1) 
public void removeMPDecoderMediaListener 
(MPDecoderMediaListener 1) 
Interface mpgj .sys.MPRendererEventGenerator 
Subinterfaces: Renderer 
public interface MPRendererEventGenerator 
Methods 

public void addMPRendere rMediaListener 
(MPRenderMediaListener 1) 

public void removeMPRendererMediaListener 
(MPRenderMediaListener 1) 
Interface mpgj. sys. Renderer 

public interface Renderer 

implements MPRendererEventGenerator 

For each decoder, the Resource Manager would have an 
installation of a class that implements the MPDecoder 
interface, described below, or a sub-interface. These decoder 
installations generate the events for different situations. The 
resource manager implementation can handle events, if 
necessary, in addition to the event handlers in the application 
(the order of which is left to the implementation). The 
MPEG-J application can receive the Event handlers as 
bytecode in the bitstream. 
Interface mpgj .sys.ResourceManager 

public interface ResourceManager 
Methods 

public void setDecPriority(MPDecoder dec, int priority) 
Overrides or changes the priority of a media elementary 
stream. 

public int getDecPriority(MPDecoder dec) 

Gets the priority of a media elementary stream. 

public void checkDynResources(int memory,int scratch, 

int processLoad) 
Gets dynamic system resource information, such as, for 
example, information about memory, scratch space and 
processor loads. 

public void checkStatResources(int grpcards.int devices, 
int plugins) 



Gets static system resource information, such as, for 
example, information about graphics cards, hardware 
devices and software plugins. 

public MPDecoder getDecoder(int nodeid) 
s Gets the media decoder for the identified node. 

public Renderer getRenderer ( ) 

Gets the renderer and readies the presentation engine to 
add events to renderer. 
Interface mpgj.sys.MPDecoderMediaListener 

public interface MPDecoderMediaListener 
extends java.util.EventListener 
Methods 

public abstract void mPDecoderMediaHandler 
!5 (MPDecoderMediaEvents event) 

Interface mpgj. sys. MPRendererMediaListener 
public interface MPRendererMediaListener 
extends java.util.EventListener 
Methods 

20 public abstract void mPRendererMediaHandler 
(MPRendererMediaEvents event) 
Class mpgj.sys.MPDecoderMediaEvents 
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java.lang.Object 

+ - - - java.utLLEventObject 



+- - mpgj.sys.MPDecoderMediaEvents 

30 

public class MPDecoderMediaEvents 
extends java.util.EventObject 
Constants and Variables 
35 public static final int DECODER_OVERFLOW 
public static final int DECODER_SYNC_£RROR 
public static final int STREAM_UNDERFLOW 
public static final int STREAM_OVERFLOW 
40 public static final int STREAM_START 
public static final int STREAM__END 
Constructors 

public MPDecoderMediaEvents(int condition) 
Methods 

public int getcondition ( ) 

The obtained condition can correspond to decoder 
overflow, sync error or end of decode. 
Class mpgj. sys .MPRendererMediaEvents 
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java.lang.Object 

+ — java.util.EventObject 

+ - -mpgj.sys. MPRendererMediaEvents 



public class MPRendererMediaEvents 
extends java.util.EventObject 
60 Constants and Variables 

public static final int DECODER_UNDERFLOW 
public static final int MISSED__FRAMES 
Constructors 

65 public MPRendererMediaEvents(int condition) 
Methods 

public int getCondition ( ) 
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The MPEG -J Media Decoders application programming 
interface specification is set forth below: 
Class mpgj.dec.MPDecoder 

public abstract class MPDecoder 
Implements MPDecoderEventGenerator and 
MPStreamEventGenerator. 

This is the base class for media decoders. 
Constructors 

public MPDecoder 
Methods 

public void start 

Starts decoding of data, 
public void stop ( ) 

Stops decoding of data, 
public void pause ( ) 

Pauses decoding of data, 
public void resume ( ) 

Resumes decoding of data, 
public void attach (int esid) 

Attaches a decoder for decoding of data, 
public void detach ( ) 
Detaches a decoder already decoding data. 
Interface mpgj. dec. SclDecoder 

Subinterfaces: SclTexture Decoder, SclVideoDecoder 
public interface ScdDecoder 

extends MPDecoderEventGenerator, MPStreamEvent- 
Generator 

This is the base interface for scalable enhancement. 
Methods 

public int getLevel ( ) 

Gets the spatial enhancement layer number. 

public int getMode ( ) 

Gets the spatial enhancement mode number. 35 

pub He boolean sptEnh(int e_level,e 13 mode) 
throws InvalidDecoderLevelException, 
InvalidDecoderModeException 

Sets the decoding mode to spatial enhancement layer and 
returns a boolean to indicate if it succeeded or not, e 13 level 40 
is the enhancement layer number, while e 13 mode is the 
subtype in that layer. 

public boolean tmpEnh(int e 13 level,int e_mode) 
throws InvalidDecoderLevelException, 
InvalidDecoderModeException 45 

Sets the decoding mode to temporal enhancement, layer 
and returns a boolean to indicate if it succeeded or not. 

public boolean snrEnh(int e_Jevel,int e_mode) 
throws InvalidDecoderLevelException, 
InvalidDecoderModeException 

Sets the decoding mode to snr enhancement layer and 
returns a boolean to indicate if it succeeded or not. 
Interface mpgj.dec.VideoDecoder 

Subinterfaces: SclVideoDecoder, Transparency 

public interface VideoDecoder 
extends MPDecoder 

Decoding of video objects. 
Interface mpgj.dec.SclVideoDecoder 

Subinterfaces: DirecDecoding, HotObject 

public interface SclVideoDecoder 
extendsVideoDecoder, SclDecoder 

Decoding of scalable video objects. 
Methods 

public boolean datapart(int e_level,int e_mode) 
Sets the decoding mode to data partitioning enhancement 
layer and returns a boolean to indicate if it succeeded or not. 
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Interface mpgj.dec.TextureDecoder 
Subinterfaces: SclTextureDecoder 
public interface TextureDecoder 

extends MPDecoder 
Decoding of image texture objects 
Interface mpgj.dec.SclTextureDecoder 
Subinterfaces: Progressivelmage 
public interface SclTextureDecoder 
extends TextureDecoder, SclDecoder 
Interface mpgji.dcc.MeshDecoder 
public interface MeshDecoder 

extends MPDecoder 
Decoding of wireframe mesh objects 
Interface mpgj.dec.FaceDecoder 
Subinterfaces: SclFaceDecoder 
public interface FaceDecoder 

extends MPDecoder 
Decoding of Face objects 
Interface mpgj.dec.SclFaceDecoder 
public interface SclFaceDecoder 

extends FaceDecoder 
This interface extends FaceDecoder. It contains methods 
to scale decoder performance. 
Methods 

public boolean pickModel(int mode) 

Sets the decoding mode and returns a boolean to indicate 
if it succeeded or not. 

The 'mode' variable indicates one of the three choices — 
default model without calibration, default model with 
calibration, and, the downloaded model. 

public boolean pickFAP(int mode) 

Sets the decoding mode and returns a boolean to indicate 
if it succeeded or not. The 'mode' variable indicates one of 
the two choices — all FAP groups, and, viseme and expres- 
sion group of FAP only. 

public boolean pickFIT(int mode) 

Sets the decoding mode and returns a boolean to indicate 
if it succeeded or not. The 'mode' variable indicates one of 
the two choices — use FIT, and, ignore FIT. 

public boolean sclBitrateDown ( ) 

Decrease decoder bitrate. 

public boolean sclcomplexity ( ) 

Decrease decoder complexity. 
Interface mpgj. dec. AudioDecoder 

public interface Audio Decoder 
extends MPDecoder 
Interface mpgj.dec.SclAudioDecoder 

public interface SclAudioDecoder 
extends AudioDecoder 

This interface extends AudioDecoder. It contains methods 
to scale decoder performance. 
Constants 

public static final int PARAMETRIC 

public static final int CELP 

public static final int AAC 
Methods 

public boolean sclBitrateUp ( ) 

Increase decoder bitrate for scalable stream. 

public boolean sclBitrateDown ( ) 

Decrease decoder bitrate for scalable stream. 

public boolean sclBandwidth (int region) 

public boolean sclComplexity ( ) 
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public boolean sclType (int type) 
Interface mpgj.dec.TTS Decoder 

public interface TTSDecoder 
extends MPDecoder 

Decoding of TTS objects. 5 
Interface mpgj.dec.StructAudioDecoder 

public interface StructAudioDecoder 
extends MPDecoder 

Decoding of structured audio objects. 

The MPEG-J Functionality application programming 10 
interface specification is set forth below: 
Interface mpgj .func.Proaressivelmage 

public interface Progressivelmage 
extends SclTextureDecoder 

Progressivelmage allows progressive refinement of qual- 15 
ity of an image under user control. The images are assumed 
to be static (still image vops). 
Methods 

public int selectProgLevel ( ) 

Select level up to which decoding of transform (DCT or 20 
wavelet) coefficients will take place. A level constitutes 
coefficients up to a certain position in scan order. 
Interface mpgj.func.HotObject 

public interface HotObject 

extends SclVideoDecoder 25 

HotObject is a interface that triggers the action of 
enhancement of a visual object provided that the object is a 
hot object. 
Methods 

public int selectHotType ( ) 30 
Selects type of enhancement (spatial, quality, temporal 
etc.). 

Interface mpgj.mnc.DirecDecoding 

public interface DirecDecoding ^ 
extends SclVideoDecoder 

DirecDecoding is an interface that allows creation of 
visual objects that are direct] onally sensitive. 
Methods 

public int selectDirec ( ) 

Selects the direction (scene orientation). A number of 
prespecified directions are allowed and selection takes place 
by clicking a mouse on hot points on the object or via a 
menu. 

Interface mpgj.func.Transparency 45 

public interface Transparency 
extends VideoDecoder 

The Transparency interface allows formation of visual 
objects with transparency information. 

Methods 50 

public int redirect ( ) 
Interface mpgj.func.TrickDecoding 

public interface TrickDecoding 
extends MPDecoder 

Selective decoding of objects for playback intrickmode 55 
(FF, FR.) under user control. TrickDecoding is an interface 
that can be used to decode audio visual objects for trick play. 
Methods 

public float fastforw (float speed) 

throws InvalidDecoderSpeedException 60 

Start decoding of data at the speed specified. The speed 
must be positive. The speed=l for normal playing. It returns 
the effective speed set. 

public float fastrev (float speed) 

throws InvalidDecoderSpeedException 65 

Stop decoding of data at the speed specified. The speed 
must be positive. The speed-1 for normal playing. 
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public void freezeDec ( ) 
Temporarily suspend decoding of data, 
public void resumeDec ( ) 

Restart decoding of data from current state of pause, 
public void loopDec ( ) 

This method allows user interactive decoding of selected 
portions of the srcstream for forward or reverse playback at 
a variety of speeds. 

public float getSpeedAndDirec ( ) 

Get the direction of decoding and speed. 
Interface mpgj.func. AdjustAudio 

public interface AdjustAudio 
extends SclAudioDecoder 

AdjustAudio allows refinement of quality of audio under 
user control. 
Methods 

public boolean incrSpeed(float percent) 

Increase speed of decoder without changing pitch. 

public boolean decrSpeed(float percent) 

Decrease speed of decoder without changing pitch. 

public boolean incrpitch(float percent) 

Increase pitch of decoder without changing speed. 

public boolean decrpitch(float percent) 

Decrease pitch of decoder without changing speed. 

The MPEG-J Device application programming interface 

specification is set forth below: 

Interface mpgj.dev.InputDevice 
public interface InputDevice 

The input device application programming interface is a 
common interface for input device drivers. The application 
would expect all of the methods of the application program- 
ming interface to be implemented. This API is modeled after 
the JAVA 3D Input Device application programming 
interface, and is intended as a starting point for discussion. 
Constants 

public static final int POLLED 

public static final int STREAMING 

These flags specify the mode in which the associated 
device works. 
Methods 

public abstract boolean initialize ( ) 
Returns true if initialization succeeds, false otherwise, 
public abstract void setProcessingMode(int mode) 
public abstract int getProcessingMode ( ) 
public abstract int getSensorCount ( ) 
Returns the number of sensor objects associated with the 
device. 

public abstract Sensor getSensor(int sensorindex) 
Returns the specified sensor 

public abstract void setNominalPositionAndOrientation ( 

) ... 

Sets the devices current position and orientation as nomi- 
nal. 

public abstract void pollAndProcessInput ( ) 
Polls the device for data and processes the received 
values. 

public abstract void processStreamlnput ( ) 
public abstract void close ( ) 
Interface mpgj.dev.AudioDevice 
public class AudioDevice 

The audio device interface is a preliminary specification 
of what would be required to incorporate a device driver for 
an audio device into an MPEG-J implementation. This API 
is modeled after the JAVA 3D AudioDevice API. 
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Constants 

public final static int HEADPHONES 

public final static int MONO„SPEAKER 

public final static int STEREO_SPEAKERS 

public final static int SURROUND_5_l 

public final static int SURROUND_7_l 
Methods 

public abstract boolean initialize ( ) 
public abstract boolean close ( ) 
public abstract void setAudioPlaybackType (int type) 
public abstract int getAudioPlaybackiype ( ) 
Set or retrieve the device type (Mono Stereo or 
Headphones) 

public abstract void setCenterEarToSpeaker (float 
distance) ' 

public abstract float getCenterEarToSpeaker ( ) 
Sets or retrieves the distance from the center ear and one 
of the speakers in the listeners environment. 

public abstract void setAngleOffsetToSpeaker (float 
angle) 

public abstract float getAngleOffeetToSpeaker ( ) 
Sets or retrieves the angle between the vectors from the 
center ear to each of the speaker transducers, 
public abstract int getTotalChannels ( ) 
public abstract int getChannelsAvailable ( ) 
public abstract int getChannelsUsedForSound ( ) 
Returns the number of channels needed to render a 
particular sound node. 

The MPEG-J Interactivity Manager application program- 
ming interface specification is set forth below: 
Interface mpgj.sys. Inter activityManager 
public interface InteractivityManager 
Ibis interface provides a centralized facility for managing 
user interactivity with the scene. 
Methods 

pub he java.lang.String getName ( ) 
public void setpickability(int id, int pickability) 
public void clearpickability(int id, int pickability) 
public void getPickability(int id, int pickability) 
public void resetPickability(int id) 
The value "id" specifies the node for every pickable 
object. There is 1 bit associated with each capability. Thus, 
it is possible to set or reset one capability at a time. The 
Mouse Events include mouse click, mouse drag, mouse 
entered, mouse exited, mouse first, mouse last, mouse 
moved, mouse pressed, and mouse released. 

As shown in FIGS. 1-5, the MPEG J application engine 
and the MPEG-4 browser are preferably implemented on a 
multimedia data processing system such as a programmed 
general purpose computer. However, the MPEG-J applica- 
tion engine and the MPEG-4 browser can also be imple- 
mented on a special purpose computer, a programmed 
microprocessor or microcontroller and peripheral integrated 
circuit elements, an ASIC or other integrated circuit, a 
digital signal processor, a hardwired electronic or logic 
circuit such as a discrete element circuit, a programmable 
logic device such as a PLD, PLA, FPGA or PAL, or the like. 
In general, any device, capable of implementing a finite state 
machine that is in turn capable of implementing the appli- 
cation programming interfaces and the various system ele- 
ments shown in FIGS. 1-5, can be used to implement the 
MPEG-J application engine and the MPEG-4 browser. 

While this invention has been described in conjunction 
with the exemplary embodiments outlined above, it is evi- 
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dent that many alternatives, modifications and variations 
will be apparent to those skilled in the art. Accordingly, the 
exemplary embodiments of the invention, as set forth above, 
are intended to be illustrative, not limiting. Various changes 
may be made without departing from the spirit and scope of 
the invention. 
What is claimed is: 

1. A multimedia browsing system, comprising: 

a multimedia browsing subsystem that generates a mul- 
timedia scene based on a scene description, the multi- 
media scene comprising at least one multimedia object 
and the scene description defining spatial and temporal 
positioning of each object within the scene; and 

a multimedia browsing control subsystem that controls 
the multimedia browsing subsystem based on a control 
program associated with the at least one multimedia 
object; 

wherein: 

the multimedia browsing control subsystem controls a 
behavior of the multimedia browsing subsystem 
based on behavior control commands of the control 
program associated with the at least one multimedia 
object; 

the multimedia browsing system is executing on a data 
processing apparatus having at least one system 
resource; 

the behavior control commands associated with the at 
least one multimedia object controllably modifying 
the multimedia scene based at least on a status of the 
one system resource; 

the multimedia browsing subsystem is controlled by 
the multimedia browsing control subsystem through 
at least one application programming interface, 
which includes at least one each of an application 
manager application programming interface, a scene 
graph application programming interface, a resource 
manager application programming interface, a media 
decoder application programming interface, a func- 
tionality application programming interface, a net- 
work application programming interface, a device 
application programming interface, and an interac- 
tivity application programming interface; 

wherein: 

the multimedia browsing control subsystem com- 
prises: 

an application controller; and 
a scene graph manager; 
the scene graph application programming interface 
connects the application controller and the scene 
graph manager; 
the scene graph manager is connected to a scene 
graph controller of the multimedia browsing sub- 
system; 

wherein the multimedia browsing subsystem further 
comprises a scene graph decoder that decodes 
encoded scene graph data, and that outputs the 
decoded scene graph data to the scene graph 
controller; 

the multimedia browsing control subsystem further 
comprises a scene encoder connected to the scene 
graph decoder of the multimedia browsing sub- 
system and controllably connectable to the scene 
graph manager; and 

the scene graph manager controls the scene encoder 
to create additional encoded scene graph data 
based at least on the multimedia scene being 
controllably modified based at least on a status of 
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the one system resource, the scene encoder out- 
putting the additional encoded scene graph data to 
the scene graph decoder. 

2. The multimedia browsing system of claim 1, wherein 
the multimedia browsing control subsystem further com- 
prises a switch that selectively connects the scene graph 
manager to one of the scene encoder and the scene graph 
controller. 

3. The multimedia browsing system according to claim 1, 
wherein the behavior control commands associated with the 
at least one multimedia object controllably degrade the 
multimedia scene based at least on a status of the one system 
resource. 

4. A method for browsing an MPEG-4 multimedia scene 
comprising at least one MPEG-4 multimedia object, com- 
prising: 

generating, using a multimedia browsing subsystem, the 
MPEG-4 multimedia scene based on an MPEG-4 scene 
description, the MPEG-4 scene description defining 
spatial and temporal positioning of each MPEG-4 mul- 
timedia object within the scene, including executing the 
MPEG-4 scene description on a data processing appa- 
ratus having at least one system resource; 

controlling the generation of the MPEG-4 multimedia 
scene based on an MPEG-J control program associated 25 
with the at least one MPEG-4 multimedia object by 
controlling the multimedia browsing subsystem using a 
multimedia browsing control subsystem, which 
includes at least one each of an application manager 
application programming interface, a scene graph 
application programming interface, a resource manager 
application programming interface, a media decoder 
application programming interface, a functionality 
application programming interface, a network applica- 
tion programming interface, a device application pro- 
gramming interface, and an interactivity application 
programming interface, including: 
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connecting the multimedia browsing subsystem to the 
multimedia browsing control subsystem through at 
least one application programming interface; 

connecting an application controller and a scene graph 
manager through the scene graph application pro- 
gramming interface; 

connecting the scene graph manager to a scene graph 
controller of the multimedia browsing subsystem; 

controllably modifying the MPEG-4 multimedia scene 
based at least on a status of the one system resource; 

decoding encoded scene graph data using a scene graph 
decoder of the multimedia browsing subsystem; 

outputting decoded scene graph data to the scene graph 
controller; 

connecting a scene graph encoder to the scene graph 
decoder of the multimedia browsing subsystem; 

controllably connecting the scene graph encoder to the 
scene graph manager; 

controlling the scene graph encoder using the scene 
graph manager to create additional encoded scene 
graph data based at least on the multimedia scene 
being controllably modified based at least on a status 
of the one system resource; and 

outputting the additional encoded scene graph data 
from the scene graph encoder to the scene graph 
decoder. 

5. The method of claim 4, further comprising: 
selectively connecting the scene graph manager to one of 

the scene encoder and the scene graph controller using 
a switch. 

6. The method according to claim 4, wherein controllably 
modifying the MPEG-4 multimedia scene includes control- 
lably degrading the MPEG-4 multimedia scene based at 
least on a status of the one system resource. 
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